🏗

Azure AI Foundry Overview

Azure AI Foundry (formerly Azure AI Studio) is Microsoft's unified platform for building, deploying, and managing enterprise AI solutions. It organizes work into Hubs and Projects.

🔧

Selecting the Right Azure AI Service

You must match a business scenario to the correct Azure AI service family. This is heavily tested.

⚙

Creating & Deploying AI Services

Azure AI resources can be deployed as multi-service resources or single-service resources.

• Multi-service resource: Single endpoint and key for multiple AI services — simpler, fewer resources to manage.
• Single-service resource: Dedicated resource per service — better isolation, service-specific tiers and limits.
• Resources require a resource group, region, pricing tier, and name.
• Some services (e.g., OpenAI) require separate approved subscriptions.

🔑

Authentication Methods

📊

Monitoring Azure AI Resources

Azure AI services integrate with Azure Monitor for metrics, logs, and alerts.

• Metrics: Request count, latency, error rate, throttled calls — available via Azure Monitor.
• Diagnostic Logs: Must be explicitly enabled. Send to Log Analytics, Storage Account, or Event Hub.
• Log Analytics Workspace: Use KQL (Kusto Query Language) to query logs.
• Alerts: Configure metric alerts for anomaly detection (e.g., error rate > 5%).
• Azure AI Foundry: Model monitoring for generative AI — monitors performance, safety, resource consumption.

🔒

Network Security for Foundry Services

• Public endpoint: Default — accessible from internet (use firewall rules + key auth).
• Virtual Network (VNet) Service Endpoint: Restricts access to specific VNet subnets.
• Private Endpoint: Private IP in your VNet — no traffic over internet; best for enterprise.
• IP Firewall Rules: Allowlist specific IP ranges for public endpoint.
• Foundry Hub networking: Configure at Hub level — Projects inherit the network configuration.

💰

Managing Costs

• Pricing tiers: Free (F0), Standard (S0), and premium tiers — F0 has usage limits (typically 5k calls/month).
• Token-based billing: Azure OpenAI bills per 1K input and output tokens separately.
• PTU (Provisioned Throughput Units): Reserved capacity for Azure OpenAI — predictable latency and cost.
• Azure Cost Management: Use budgets and alerts to prevent overspend.
• Content caching: Prompt caching can reduce OpenAI costs for repeated prompts.

🔄

CI/CD Integration

Integrate Azure AI resources into automated pipelines using:

• Azure DevOps / GitHub Actions: Deploy AI resources via ARM templates, Bicep, or Terraform.
• Azure AI Foundry SDK: Use azure-ai-projects Python SDK to automate model deployment in pipelines.
• Model versioning: Track model versions; use deployment slots for staged rollouts.
• Prompt Flow: CI/CD for LLM applications — version, test, and deploy prompt flows as endpoints.

⚖

Microsoft's 6 Responsible AI Principles

🛡

Azure AI Content Safety Features

📋

Responsible AI Governance Framework

Designing a governance framework involves:

1. Impact Assessment: Identify potential harms before deployment
2. Model Cards: Document model capabilities, limitations, and intended uses
3. Human-in-the-loop: Define when human review is required
4. Monitoring: Continuously monitor for fairness drift and harmful outputs
5. Incident Response: Define escalation path for AI failures
6. Data Governance: Document training data sources and consent

🏗

Azure AI Foundry Setup

To build a GenAI solution in Foundry:

1. Create a Foundry Hub (top-level, manages shared infra + security)
2. Create a Foundry Project under the Hub (workspace for the solution)
3. Deploy a model within the project (GPT-4o, Llama 3, etc.)
4. Build with Prompt Flow or SDK
5. Evaluate with built-in evaluators
6. Deploy as a managed online endpoint

🔄

RAG — Retrieval Augmented Generation

RAG grounds model responses in your private data. Key components:

📊

Prompt Flow

Prompt Flow is an LLM-ops tool for building, testing, and deploying LLM applications.

• Flow types: Standard (general), Chat (multi-turn), Evaluation (scoring)
• Nodes: LLM, Python, Prompt, Tool — connected as a DAG
• Variants: Test different prompt versions in parallel
• Evaluators: Built-in metrics — groundedness, relevance, coherence, fluency, similarity
• Deployment: Deploy flows as REST endpoints with auto-scaling

📐

Prompt Engineering Techniques

🤖

Azure OpenAI Models

⚙

Key Parameters for Generation Control

🖼

DALL-E & Multimodal

DALL-E 3 generates images from text via the Azure OpenAI endpoint:

For GPT-4o vision, pass image as base64 or URL in the user message content array with type: "image_url".

🎛

Fine-tuning

Fine-tuning adapts a base model on your domain-specific data:

• Supported models: GPT-4o mini, GPT-3.5 Turbo (check current availability)
• Training format: JSONL with {"messages": [...]} pairs
• Minimum dataset: 10 examples (recommend 50–100+ for good results)
• When to fine-tune: Consistent style/format, domain terminology, not for adding new knowledge (use RAG instead)
• Cost: Training cost + higher inference cost vs base model

📈

Model Evaluation Metrics

🔍

Tracing & Feedback

• Azure AI Foundry tracing: Track each step in a prompt flow — inputs, outputs, latency per node.
• OpenTelemetry integration: Export traces to Azure Monitor / Application Insights.
• Feedback collection: Collect thumbs up/down ratings; feed back into evaluation datasets.
• Model reflection: Use a secondary LLM call to self-critique the primary response.

🌊

Deployment & Scalability

• Standard deployment: Pay-per-token, auto-scales to Azure capacity.
• PTU deployment: Provisioned Throughput Units — reserved compute, consistent latency, hourly billing.
• Container deployment: Deploy Foundry models to ACI, AKS, or edge (IoT) using containers.
• Multi-model orchestration: Route different request types to specialized models (e.g., GPT-4o for complex, GPT-4o mini for simple).

🤖

What is an AI Agent?

An AI agent is an autonomous system that perceives its environment, makes decisions, and takes actions to achieve goals — using LLMs as the reasoning engine.

• Core loop: Perceive → Reason → Act (ReAct pattern)
• Tools: Agents call external tools/APIs — code interpreter, search, databases, custom APIs
• Memory: Short-term (conversation context), long-term (vector stores), episodic
• Planning: Break complex tasks into sub-tasks; execute step-by-step
• Reflection: Self-evaluate outputs and retry if needed

🏗

Microsoft Foundry Agent Service

The Azure AI Foundry Agent Service provides a managed platform for building and deploying agents:

• Create agents via Foundry portal or SDK with instructions, model, tools, and thread management.
• Built-in tools: Code Interpreter, File Search (RAG), Function Calling, Azure AI Search, Bing Search.
• Threads: Manage conversation state; messages persist in thread context.
• Runs: Execute agent on a thread; poll for completion or use streaming.

🔗

Multi-Agent Orchestration

• Microsoft Agent Framework (Semantic Kernel / AutoGen): Build complex workflows with multiple specialized agents.
• Orchestrator agent: Routes tasks to specialized sub-agents.
• Sub-agents: Each handles a specific domain (e.g., SQL agent, document agent, search agent).
• Autonomous capabilities: Agents can trigger other agents; use event-driven patterns.
• Testing: Test agents with diverse scenarios; evaluate tool use accuracy and goal completion.

🖼

Azure Vision in Foundry Tools — Capabilities

📄

OCR & Read API

The Read API (Azure Vision 4.0 Read) extracts text from images and documents:

• Input formats: JPEG, PNG, BMP, TIFF, PDF — max 50 MB, max 10,000 pages
• Languages: 164+ languages supported
• Handwriting: Detect and extract handwritten text
• Response: Returns text lines, words, bounding polygons, confidence scores
• Async operation: For large documents — submit and poll with operation ID

🎯

Custom Vision — Classification vs Detection

Training workflow: Upload images → Label → Train → Evaluate (Precision/Recall/AP) → Publish → Consume via prediction URL

🎬

Video Analysis Services

Video Indexer insights include: transcript, OCR, keywords, labels, scenes, shots, keyframes, faces, named people, brands, sentiments.

📝

Azure AI Language — Text Analytics Features

🌍

Azure Translator in Foundry Tools

• Text translation: Translate text between 100+ languages in a single API call
• Document translation: Translate entire documents while preserving layout — async batch operation
• Transliteration: Convert script without translating (e.g., Arabic → Latin letters)
• Language detection: Auto-detect source language
• Dictionary lookup: Find alternative translations and word usage examples
• Custom Translator: Fine-tune translation for domain-specific terminology (e.g., legal, medical)

🎙

Azure Speech in Foundry Tools

🎨

SSML — Speech Synthesis Markup Language

SSML allows fine-grained control over TTS output:

SSML elements: <voice> (select voice), <prosody> (rate/pitch/volume), <break> (pause), <emphasis> (stress), <say-as> (dates, ordinals), <phoneme> (pronunciation).

🔊

Audio Formats for Speech

🧠

Conversational Language Understanding (CLU)

CLU (successor to LUIS) builds custom intent/entity recognition models:

Workflow: Create project → Add intents/entities → Add utterances → Train → Evaluate (F1/Precision/Recall) → Deploy → Consume

❓

Custom Question Answering (QnA)

Build FAQ-style knowledge bases that answer questions from documents:

• Sources: URLs (FAQ pages), files (PDF, DOCX, TXT, TSV), manual Q&A pairs
• Multi-turn conversations: Add follow-up prompts to create guided conversation flows
• Chit-chat: Add personality responses (Professional, Friendly, Witty, etc.)
• Alternate phrasings: Add variant ways to ask the same question
• Active learning: System suggests improvements based on low-confidence answers
• Multi-language: Set project language or use per-document language detection
• Export: Export as TSV or JSON for backup/migration

🌐

Custom Translator

• Train custom translation models for domain-specific terminology
• Training data: Parallel sentence pairs (source + target), bilingual documents
• Min training data: 1,000 sentence pairs (recommend 10,000+)
• Publish to custom category; call via category parameter in Translator API
• Iterative improvement: Add more data, retrain, compare BLEU scores

🔍

Core Architecture

Azure AI Search (formerly Cognitive Search) has 4 main components:

⚡

Built-in Cognitive Skills

📦

Custom Skills

Extend skillsets with custom logic:

• Custom Web API skill: Call any HTTPS endpoint that accepts the skill contract (JSON in/out)
• Azure Function skill: Easiest pattern — deploy Azure Function, register as custom skill
• Azure ML skill: Call Azure ML endpoint for custom model inference
• Input/Output schema: Must match skillset contract — array of values with document key

🔎

Query Types & Syntax

📋

Prebuilt Models

🎨

Custom Models

Labeling workflow: Upload samples to Azure Blob → Label in Document Intelligence Studio → Train → Evaluate (F1/accuracy per field) → Publish → Consume

📁

Supported Formats & Limits

🌐

Azure Content Understanding Overview

Azure Content Understanding (in Foundry Tools) is a new multimodal extraction service that goes beyond Document Intelligence — it processes documents, images, videos, and audio to extract structured insights.